Fusion Algorithm for Vidar Traffic Surveillance System

ABSTRACT

This invention is related to a fusion algorithm for a video-Doppler-radar (Vidar) traffic surveillance system comprising of (1) a robust matching algorithm which iteratively matches the information from a video camera and multiple Doppler radars corresponding to a same moving vehicle, and (2) a stochastic algorithm which fuses the matched information from the video camera and Doppler radars to derive the vehicle velocity and range information.

TECHNICAL FIELD

This invention relates to a fusion algorithm for a Vidar trafficsurveillance system.

BACKGROUND OF THE INVENTION

A traditional radar based traffic surveillance system uses a Dopplerradar for vehicle speed monitoring which measures a vehicle speed atline-of-sight (LOS). In FIG. 1, the speed of an approaching (or aleaving) vehicle is calculated in terms of Doppler frequency f_(D) by

$\begin{matrix}{v_{t} = \frac{f_{D}}{K\; {\cos \left( \varphi_{t} \right)}}} & (1)\end{matrix}$

where K is a Doppler frequency conversion constant and φ_(t) is calledthe Doppler cone angle or simply the Doppler angle. Although a Dopplerradar based system has an advantage of a long detection range, there areseveral difficulties associated with the traditional radar based system,including (1) the Doppler radar beam angle is too large to preciselylocate vehicles within the radar beam; (2) the angle between the vehiclemoving direction and the LOS, is unknown and therefore, needs to besmall enough for a reasonable speed estimation accuracy; (3) since allvelocity vectors on the equal-Doppler cone in FIG. 1 will generate asame speed, the Doppler radar cannot differentiate the vehicles with asame speed but different directions defined by the same equal-Dopplercone. Therefore, no precise target location information can be derivedin a traditional Doppler radar based traffic surveillance system.

Video Camera Based Traffic Surveillance Systems

A video camera based traffic surveillance system uses a video camera tocapture a traffic scene and relies on computer vision techniques toindirectly calculate vehicle speeds. Precise vehicle locations can beidentified. However, since no direct speed measurements are availableand the camera has a finite number of pixels, the video camera basedtraffic surveillance system can be used only in a short distanceapplication.

Video-Doppler-Radar (Vidar) Traffic Surveillance Systems

A video-Doppler-radar (Vidar) traffic surveillance system combines boththe Doppler radar based system and the video based system into a uniquesystem to preserve the advantages of both systems and overcome theshortcomings of both systems. A patent application on Vidar trafficsurveillance system has been filed by the first author, PatentApplication No. 12266227.

A Vidar traffic surveillance system may include a first movable Dopplerradar to generate a first radar beam along the direction of a firstmotion ray, a second movable Doppler radar to generate a second radarbeam along the direction of a second motion ray, a third fixed Dopplerradar to generate a third radar beam along a direction ray, a videocamera to serve as an information fusion platform by intersecting thefirst and second radar motion rays with the camera virtual image plane,a data processing device to process Doppler radar and video information,a tracking device to continuously point the surveillance system to themoving vehicle, and a recording device to continuously record thecomplete information of the moving vehicle.

Robustly matching information from a video camera and multiple Dopplerradars is a prerequisite for information fusion in a Vidar trafficsurveillance system. However, because of the different modalities ofvideo and Doppler radar sensors, matching information from a videocamera and Doppler radars is very difficult. Due to the specialvideo-radar geometry introduced in Vidar, correctly matching between avideo sequence and Doppler signals is possible. This invention describesa robust algorithm to match video signals and Doppler radar signals, andan algorithm to fuse the matched video and Doppler radar signals.

SUMMARY

A fusion algorithm for a Vidar traffic surveillance system may includethe following steps: (1) deriving Doppler angles from a video sequence;(2) generating estimated Doppler signals from estimated Doppler angles;(3) matching estimated Doppler signals to the measured Doppler signalsof two moving Doppler radars; (4) finding the best match between theestimated and measured Doppler signals; (5) forming a three-scan,range-Doppler geometry from the stationary Doppler radar and estimatedDoppler angles; (6) matching video signals to stationary Doppler radarsignals; (7) fusing the matched video and Doppler radar signals togenerate moving vehicle velocity and range information.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention may be understood by reference to the followingdescription taken in conjunction with the accompanying drawings, inwhich, like reference numerals identify like elements, and in which:

FIG. 1 illustrates the fundamental of a Doppler radar for speedmeasuring;

FIG. 2 illustrates the functional flow chart of the fusion algorithm;

FIG. 3 illustrates the layout of the Vidar sensor suite;

FIG. 4 illustrates the sensing geometry of the Vidar trafficsurveillance system; and

FIG. 5 illustrates a three-scan geometry for fusing video and Dopplerradar signals.

DETAILED DESCRIPTION

The functional flow chart of the algorithm is shown in FIG. 2. In thefollowing, we will provide detailed description of the invention.

BRIEF DESCRIPTION OF VIDAR SENSOR SUITE

FIG. 3 shows the layout of the Vidar sensor suite 201 where 202—a firstmoving Doppler radar, 203—a second moving Doppler radar, 204—a fixed orstationary Doppler radar, 205—a fixed or stationary video camera, 206—adata processing device, such as a computer, laptop, personal computer,PDA or other such device, and 207—a data recording device, such as ahard drive, a flash drive or other such device. FIG. 3 also indicatesthe sensing geometry where 208—the camera virtual image plane of thevideo camera 205, 212—a first moving Doppler radar motion ray, 213—asecond moving Doppler radar motion ray, 214—a radar direction rayconnecting the Vidar device apparatus 201 to a moving vehicle 215,209—the intersection of the first Doppler radar motion ray 212 with thevirtual image plane 208, 210—the intersection of the second Dopplerradar motion ray 213 with the virtual image plane 208, 211—theintersection of the ray connecting the Vidar apparatus 201 and themoving vehicle 215 with the virtual image plane 208, and 215 a movingvehicle. The first and second Doppler radars 202, 203 in the Vidarapparatus 201 may be moved in such a way that the vehicle 215 is locatedin one side of both moving radar motion rays 212 and 213 withsufficiently large angles θ_(τ1) and θ_(τ2). The first and secondDoppler radars 202, 203 in the Vidar apparatus 201 may be extended orretracted or moved side to side as illustrated in FIG. 3 by a motor (notshown) which may be a DC or stepper motor or other movement device andmay be moved on sliding tracks (not shown). An optical encoder (notshown) may be mounted on the shaft of the motor, so the sliding speedsof the Doppler radars (υ_(τ1) and υ_(τ2) in FIG. 3) may bepredetermined. The sliding track orientation angles (θ_(τ1) and θ_(τ2)in FIG. 3) may also be predetermined. Using a calibration method, theintersections 209 and 210 of the first and second motion rays 212, 213with the virtual image planes 208 may be predetermined as well.

Derive Doppler Angles from a Video Sequence

The objective of this step (step 105 in FIG. 2) is to derive Dopplerangle pairs, {θ_(τ) _(1k) , θ_(τ) _(2k) } as indicated in FIG. 4 wherethe subscript k is suppressed, from an image sequence. Assume thevehicle location on the image is q_(k)=[u_(k), υ_(k)], as shown in FIG.4. The vector from O to q_(k) may be defined as Oq _(k)=[u_(k), υ_(k),f] where f is the camera focal length, and the vectors from O to C₁ andC₂ may be given by: OC ₁=[u_(c1), υ_(c1), f] and OC ₂=[u_(c2), υ_(c2),f]. The Doppler angles may be estimated in step 105 by

$\begin{matrix}{{\hat{\theta}}_{r_{1k}} = {\cos^{- 1}\frac{{\underset{\_}{Oq}}_{k} \cdot {\underset{\_}{OC}}_{1}}{{{\underset{\_}{Oq}}_{k}}{{\underset{\_}{OC}}_{1}}}}} & (2) \\{\mspace{40mu} {{= \frac{{u_{k}u_{c\; 1}} + {v_{k}v_{c\; 1}} + f^{2}}{\sqrt{u_{k}^{2} + v_{k}^{2} + f^{2}}\sqrt{u_{c\; 1}^{2} + v_{c\; 1}^{2} + f^{2}}}}{and}}} & (3) \\{{\hat{\theta}}_{r_{2k}} = {\cos^{- 1}\frac{{\underset{\_}{Oq}}_{k} \cdot {\underset{\_}{OC}}_{2}}{{{\underset{\_}{Oq}}_{k}}{{\underset{\_}{OC}}_{2}}}}} & (4) \\{\mspace{40mu} {= {\frac{{u_{k}u_{c\; 2}} + {v_{k}v_{c\; 2}} + f^{2}}{\sqrt{u_{k}^{2} + v_{k}^{2} + f^{2}}\sqrt{u_{c\; 2}^{2} + v_{c\; 2}^{2} + f^{2}}}.}}} & (5)\end{matrix}$

Match Video Signals to Moving Radar Signals

Referring to FIG. 4, the Doppler angles may be related to the Dopplersignals of the moving Doppler radars. For the first moving Dopplerradar, the following holds

f _(D) _(k) ¹ =K ₁υ_(τ) _(1k) cos(θ_(τ) _(1k) )+K ₁υ_(t) _(k)cos(φ_(k))  (6)

where υ_(t) _(k) cos(φ_(k)) may be provided by the stationary Dopplerradar via

f _(D) _(k) ³ =K ₃υ_(t) _(k) cos(φ_(k)).  (7)

Since the motion of the first moving Doppler radar is known as

υ_(τ) _(1k) =a ₁ cos(ωt _(k)+φ₁),  (8)

we have

$\begin{matrix}{f_{D_{k}}^{1} = {{K_{1}a_{1}{\cos \left( \theta_{r_{1k}} \right)}{\cos \left( {{\omega \; t_{k}} + \psi_{1}} \right)}} + {\frac{K_{1}}{K_{3}}f_{D_{k}}^{3}}}} & (9) \\{\mspace{40mu} {{= {{A_{1k}{\cos \left( {{\omega \; t_{k}} + \psi_{1}} \right)}} + {B_{1k}f_{D_{k}}^{3}}}}{where}}} & (10) \\{A_{1k} = {{K_{1}a_{1}{\cos \left( \theta_{r_{1k}} \right)}\mspace{14mu} {and}\mspace{14mu} B_{1k}} = {\frac{K_{1}}{K_{3}}.}}} & (11)\end{matrix}$

A similar equation may be derived for the second moving Doppler radar

$\begin{matrix}{{f_{D_{k}}^{2} = {{A_{2k}{\cos \left( {{\omega \; t_{k}} + \psi_{2}} \right)}} + {B_{2}f_{D_{k}}^{3}}}}{where}} & (12) \\{A_{2k} = {{K_{2}a_{2}{\cos \left( \theta_{r_{2k}} \right)}\mspace{14mu} {and}\mspace{14mu} B_{2k}} = \frac{K_{2}}{K_{3}}}} & (13)\end{matrix}$

and a₁, a₂, K₁, K₂, K₃, φ₁, and φ₂ are all known from calibration. GivenDoppler angle estimates, {circumflex over (θ)}_(τ) _(1k) and {circumflexover (θ)}_(τ) _(1k) , we have

Â _(1k) =K ₁ a ₁ cos({circumflex over (θ)}_(τ) _(1k) ) and Â _(2k) =K ₂a ₂ cos({circumflex over (θ)}_(τ) _(2k) ).  (14)

Within a predefined time window, cosine signals (Doppler signals) may begenerated as

Â _(1k) cos(ωt+φ ₁) and Â _(2k) cos(ωt+φ ₂),t _(k) −L≦t≦t _(k)  (15)

where L is the window length. It is straightforward to match estimatedcosine signals to the measured Doppler signals in a single vehicle caseusing a least square method which is performed in step 106 of FIG. 2.For a multiple vehicles case, a multiple hypothesis test may be needed.

From the video camera, multiple pairs of Doppler angles are estimated:

{θ̂_(r_(1_(k)))^(i), θ̂_(r_(2_(k)))^(i)},  i = 1, …  , N

where N is the number of vehicles, which in turn generate multiplecosine signals as

Â _(1k) ^(i) cos(ωt+φ ₁) and Â _(2k) ^(i) cos(ωt+φ ₂),i=1, . . . , N.

Using multiple hypothesis testing, the moving radar Doppler data set {D₁^(i), D₂ ^(i)} corresponding to

{θ̂_(r_(1_(k)))^(i), θ̂_(r_(2_(k)))^(i)}

may be identified (also in step 106 of FIG. 2), from which a set of newestimates may be derived:

${{{\overset{\hat{\_}}{A}}_{1k}^{i}{\cos \left( {{\omega \; t} + \phi_{1}} \right)}} + {{\overset{\hat{\_}}{f}}_{1_{D_{k}}}^{3i}\mspace{14mu} {and}\mspace{14mu} {\overset{\hat{\_}}{A}}_{2k}^{i}{\cos \left( {{\omega \; t} + \psi_{2}} \right)}} + {\overset{\hat{\_}}{f}}_{2_{D_{k}}}^{3i}},{i = 1},\ldots \mspace{14mu},{N.}$

Combing

${\overset{\hat{\_}}{f}}_{1_{D_{k}}}^{3i},{{\overset{\hat{\_}}{f}}_{2_{D_{k}}}^{3i}\mspace{14mu} {and}\mspace{14mu} {\overset{\hat{\_}}{f}}_{D_{k}}^{3i}}$

from three Doppler radars, a more accurate Doppler frequency of the ithvehicle may be determined.

Match Video Signals to Stationary Radar Signals

When two vehicles are close to each other, Doppler angles alone cannotset them apart. The stationary Doppler radar signals should provideadditional information about their speeds. In general, it is relativelymore accurate for a camera to measure an angle than derive a velocity.On the other hand, it is relatively more accurate for a Doppler radar tomeasure a velocity than derive an angle. The contribution of thisinvention is to robustly tie together the angle information from a videocamera and the Doppler (velocity) information from a Doppler radar. Inthis invention, we will match angle rates from video signals tostationary Doppler radar signals via a unique three-scan geometry.

A three-scan geometry is shown in FIG. 5, where

$\begin{matrix}{{{\Delta\theta}_{k}^{i} = {\cos^{- 1}\frac{{\underset{\_}{Oq}}_{k}^{i} \cdot {\underset{\_}{Oq}}_{k + 1}^{i}}{{{\underset{\_}{Oq}}_{k}^{i}}{{\underset{\_}{Oq}}_{k + 1}^{i}}}}}{and}\text{}{{\Delta\theta}_{k + 1}^{i} = {\cos^{- 1}\frac{{\underset{\_}{Oq}}_{k + 1}^{i} \cdot {\underset{\_}{Oq}}_{k + 2}^{i}}{{{\underset{\_}{Oq}}_{k + 1}^{i}}{{\underset{\_}{Oq}}_{k + 2}^{i}}}}}} & (16)\end{matrix}$

where Oq _(k) ^(i)=[u_(k) ^(i), τ_(k) ^(i), f] and Oq _(k+1)^(i)=[u_(k+1) ^(i), υ_(k+1) ^(i), f] are the locations of the ithvehicle on the image plane. Assume a constant velocity model, i.e.,υ_(t) _(k) ^(i)=υ_(t) _(k+1) ^(i)=|{dot over (X)} _(k) ^(i)|. Alsoassume that Doppler frequencies f³ ^(i) _(D) _(k) and f³ ^(i) _(D)_(k+1) are provided by the stationary Doppler radar. We then have

$\begin{matrix}{\Delta_{k}^{i} = {{T\frac{f_{D_{k}}^{3^{i}}}{K_{3}}\mspace{14mu} {and}\mspace{14mu} \Delta_{k + 1}^{i}} = {T{\frac{f_{D_{k + 1}}^{3^{i}}}{K_{3}}.}}}} & (17)\end{matrix}$

Using the cosine law, we have the constrained equation for thethree-scan geometry (step 107 in FIG. 2) as

(a ¹+Δ_(k) ^(i))²+(a ^(i))²−2(a ^(i)+Δ_(k) ^(j))a ^(i) cos(Δθ_(k)^(i))=(a ^(i)−Δ_(k+1) ^(i))²+(a ^(i))²−2(a ^(i)−Δ_(k+1) ^(i))a ^(i)cos(Δθ_(k+1) ^(i)).  (18)

Solving the following equation for

(a ^(i))²[2 cos(Δθ_(k+1) ^(i))−2 cos(Δθ_(k) ^(i))]+a ^(i)[2Δ_(k)^(i)+2Δ_(k+1) ^(i)−2Δ_(k) ^(i) cos(Δθ_(k) ^(i))−2Δ_(k+1) ^(i)cos(Δθ_(k+1) ^(i))]+(Δ_(k) ^(i))²−(Δ_(k+1) ^(i))²=0  (19)

we may find the range from the Vidar device to the vehicle which isperformed in step 108 of FIG. 2. Similarly,

$\begin{matrix}{b^{i} = {{a^{i} - {T\frac{f_{D_{k + 1}}^{3^{i}}}{K_{3}}\mspace{14mu} {and}\mspace{14mu} c^{i}}} = {a^{i} + {T{\frac{f_{D_{k}}^{3^{i}}}{K_{3}}.}}}}} & (20)\end{matrix}$

The criterion for matching video signals to stationary Doppler radarsignals becomes validating the following equation. Given an arbitraryDoppler signal pair from the stationary Doppler radar, say f³ ^(j) _(D)_(k) and f³ ^(j) _(D) _(k+1) , if it matches the video signals, thefollowing equation should be satisfied

(a ^(i))²[2 cos(Δθ_(k+1) ^(i))−2 cos(Δθ_(k) ^(i))]+a ^(i)[2Δ_(k)^(j)+2Δ_(k+1) ^(j)−2Δ_(k) ^(j) cos(Δθ_(k) ^(i))−2Δ_(k+1) ^(j)cos(Δθ_(k+1) ^(i))]Δ(Δ_(k) ^(j))²−(Δ_(k+1) ^(j))²=0.  (21)

Fusion of Video and Doppler Signals

Once the matched video and Doppler radar signals are found, they are fedinto a stochastic model for fusion, which is performed in step 109 ofFIG. 2.

Assume the kinematics of the ith vehicle satisfy a stochastic constantvelocity (CV) model

$\begin{matrix}{{\begin{bmatrix}\underset{\_}{X} \\\underset{\_}{\overset{.}{X}}\end{bmatrix}_{k + 1}^{i} = {{\begin{bmatrix}I & {IT} \\0 & I\end{bmatrix}\begin{bmatrix}\underset{\_}{X} \\\underset{\_}{\overset{.}{X}}\end{bmatrix}}_{k}^{i} + {\begin{bmatrix}{\frac{1}{2}{IT}^{2}} \\I\end{bmatrix}{\underset{\_}{\rho}}_{k}^{i}}}},{\left. {\underset{\_}{\rho}}_{k}^{i} \right.\sim{N\left( {\underset{\_}{0},Q_{k}^{i}} \right)}}} & (22)\end{matrix}$

where X _(k) ^(i)=[x^(i), y¹, z^(i)]_(k) is the ith vehicle's 3Dcoordinate. The positional measurement equation may be

$\begin{matrix}{0 = {{\left( {{\frac{\Delta^{i}f_{D_{k}}^{13}}{\Delta^{i}f_{D_{k}}^{23}}v_{r_{2{xk}}}} - v_{r_{1{xk}}}} \right)x_{k}^{i}} + {\left( {{\frac{\Delta^{i}f_{D_{k}}^{13}}{\Delta^{i}f_{D_{k}}^{23}}v_{r_{2{yk}}}} - v_{r_{1{yk}}}} \right)y_{k}^{i}} + {\left( {{\frac{\Delta^{i}f_{D_{k}}^{13}}{\Delta^{i}f_{D_{k}}^{23}}v_{r_{2{zk}}}} - v_{r_{1{zk}}}} \right)z_{k}^{i}}}} & (23) \\{\mspace{11mu} {= {{\begin{bmatrix}{{{\frac{\Delta^{i}f_{D_{k}}^{13}}{\Delta^{i}f_{D_{k}}^{23}}v_{r_{2{xk}}}} - v_{r_{1{xk}}}},} \\{{{\frac{\Delta^{i}f_{D_{k}}^{13}}{\Delta^{i}f_{D_{k}}^{23}}v_{r_{2{xk}}}} - v_{r_{1{xk}}}},{{\frac{\Delta^{i}f_{D_{k}}^{13}}{\Delta^{i}f_{D_{k}}^{23}}v_{r_{2{xk}}}} - v_{r_{1{xk}}}}}\end{bmatrix}{\underset{\_}{X}}_{k}^{i}} + {\underset{\_}{\gamma}}_{xk}^{i}}}} & (24) \\{\mspace{11mu} {= {{\left( {\underset{\_}{v}}_{r_{12_{k}}}^{i} \right)^{T}{\underset{\_}{X}}_{k}^{i}} + {{\underset{\_}{\gamma}}_{x_{k}}^{i}{{\left. {\underset{\_}{\gamma}}_{x_{k}}^{i} \right.\sim{N\left( {\underset{\_}{0},R_{x_{k}}^{i}} \right)}}.}}}}} & (25)\end{matrix}$

The velocity measurement equation may be established as

$\begin{matrix}{f_{D_{k}}^{3^{i}} = {{{\overset{\_}{u}}_{k}{\overset{.}{x}}_{k}^{i}} + {{\overset{\_}{v}}_{k}{\overset{.}{y}}_{k}^{i}} + {\overset{\_}{f}{\overset{.}{z}}_{k}^{i}} + {\underset{\_}{\gamma}}_{{\overset{.}{x}}_{k}}^{i}}} & (26) \\{\mspace{40mu} {= {{\left\lbrack {{\overset{\_}{u}}_{k},{\overset{\_}{v}}_{k},\overset{\_}{f}} \right\rbrack {\underset{\_}{\overset{.}{X}}}_{k}^{i}} + {\underset{\_}{\gamma}}_{{\overset{.}{x}}_{k}}^{i}}}} & (27) \\{\mspace{40mu} {{= {{{\overset{\_}{\underset{\_}{oq}}}_{k}^{T}{\overset{.}{\underset{\_}{X}}}_{k}^{i}} + {\underset{\_}{\gamma}}_{x_{k}}^{i}}}{where}}} & (28) \\{{{{\overset{\_}{u}}_{k} = {K_{3}\frac{- u_{k}}{\sqrt{u_{k}^{2} + v_{k}^{2} + f^{2}}}}},{{\overset{\_}{v}}_{k} = {K_{3}\frac{- v_{k}}{\sqrt{u_{k}^{2} + v_{k}^{2} + f^{2}}}\mspace{14mu} {and}}}}{\overset{\_}{f} = {K_{3}{\frac{- f}{\sqrt{u_{k}^{2} + v_{k}^{2} + f^{2}}}.}}}} & (29)\end{matrix}$

Eqs. (22), (25) and (28) form a stochastic system for vehicleinformation fusion and an extended Kalman filter may be used to estimatethe position and velocity of the vehicle. For a CV model, minimum twoscans may be needed and for a constant acceleration (CA) model minimumthree scans may be needed to converge.

1. A method of fusing video signals and Doppler radar signals forestimating moving vehicle velocity and range information, comprising thesteps of: a. matching said video signals to said Doppler radar signals;and b. fusing the matched said video signals and said radar signals toderive said velocity and range information of said vehicle.
 2. A methodof fusing video signals and Doppler radar signals as recited in claim 1,wherein the method estimates Doppler angles from said video signals. 3.A method of fusing video signals and Doppler radar signals as recited inclaim 1, wherein the method estimates Doppler signals from said Dopplerangles.
 4. A method of fusing video signals and Doppler radar signals asrecited in claim 1, wherein the method matches said Doppler signals tomeasured Doppler signals from moving Doppler radars.
 5. A method offusing video signals and Doppler radar signals as recited in claim 1,wherein the method forms a multiple scan geometry from said videosignals and said Doppler radar signals.
 6. A method of fusing videosignals and Doppler radar signals as recited in claim 1, wherein themethod matches said video signals to measured Doppler signals fromstationary Doppler radar.
 7. A method of fusing video signals andDoppler radar signals as recited in claim 1, wherein the method forms astochastic model for said video signals and said Doppler radar signals.8. A method of fusing video signals and Doppler radar signals as recitedin claim 1, wherein the method estimates said the velocity and rangeinformation from said video signals and said Doppler radar signals usingsaid stochastic model.